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The report examines whether increasing competition from charter schools has a caus- 
al effect on the achievement of public school students in New York City, using a three- 
year longitudinal database of student test scores. As a measure of competition, it consid- 
ers the percentage of students who left a public school for a charter school in the prior 
year. The statistical analysis suggests that increasing competition has no statistically sig- 
nificant impact on math test scores, but that it has small positive effects on language 
scores. The report does not conclusively demonstrate that the results are explained by in- 
creasing competitive pressure on public school administrators; they may also be ex- 
plained by shifting peer quality or declining short-run class sizes in public schools. 



Summary of Review 




Review 



I. Introduction 

Quantitative studies that address the impact 
of charter schools usually focus on two ques- 
tions. First, does attending a charter school 
rather than a public school cause students’ 
outcomes to increase? Second, does the entry 
of charter schools in local schooling markets 
cause students’ outcomes to increase, even if 
they remain in public schoolsl One potential 
mechanism by which public school achieve- 
ment could rise is market competition. The 
loss (or threatened loss) of students to charter 
schools would, perhaps, increase incentives 
of public school administrators and teachers 
to work differently or harder.' 

The report. Everyone Wins: How Charter 
Schools Benefit All New York City Public School 
Students^ focuses on the second question. In 
particular, it attempts to estimate the causal 
relationship between a measure of competi- 
tion from charter schools and student test 
scores in public schools. As a measure of 
competition facing public schools, it uses the 
percentage of students in a given public 
school who left for a charter school in the 
prior year. It employs three years of longitu- 
dinal data from New York City public 
schools and analyzes test scores in mathemat- 
ics and English Language Arts (ELA). The 
report posits that an increasing percentage of 
students leaving for charter schools accurate- 
ly proxies an increasing amount of pressure 
on public school administrators to “com- 
pete,” improve test scores, and staunch the 
flow of students to charter schools. The re- 
port applies a common statistical approach — 
a student fixed-effects regression — to control 
for student variables that could bias the anal- 
ysis. Overall, it concludes that increasing 
competition produces no effects on math test 
scores and small positive effects on ELA 



scores, but that such effects are slightly larger 
among public school students with lower le- 
vels of baseline achievement (the exact esti- 
mates are described in section II). 

Overall, the analysis does not make a strong 
case that these effects exclusively reflect the 
causal impact of greater competition. Even 
applying the fixed-effects analysis, it is 
plausible that an increasing percentage of 
students leaving for charter schools could 
indicate: (1) declining short-run class sizes 
in public schools, and (2) shifts in the peer 
quality of students remaining in public 
schools. Either could explain the slightly 
positive effects of ELA scores. The report 
does not attempt to empirically distinguish 
among these explanations, although it would 
be necessary to do so before drawing strong 
conclusions about the competitive effects of 
charter school entry. 

II. Findings and Conclusions 
OF THE Report 

The report’s objective is to identify a causal 
link between competition from charter 
schools and the test scores of public school 
students in New York City. Of course, com- 
petition is a diffuse and hard-to-measure 
concept. To address this, the author identi- 
fies a proxy or “stand-in” variable to meas- 
ure its intensity. Specifically, competition is 
defined as “the percentage of students in a 
[public] school who left for a charter school 
at the end of the previous year” (p. 5). Pub- 
lic schools with a greater proportion of 
“charter-leavers” are presumed to face 
greater competitive pressure to improve the 
test scores of students in the current school 
year. The average public school in this 
study’s sample lost 0.2 percent of its stu- 
dents to charter schools (p. 6). 
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The paper conducts statistical analyses, de- 
scribed in a later section, to ascertain the 
empirical association between the competi- 
tion proxy and student test scores. To sum- 
marize these results, it is convenient to state 
the effect of a 1 percentage point increase in 
the competition proxy variable: the percen- 
tage of public school students who left for a 
charter school in the prior year: 

• On average, a 1 percentage point in- 
crease in the percentage of students who 
left for a charter school has no statisti- 
cally significant effect on mathematics 
test scores. That is, the estimated math 
effect cannot be statistically distin- 
guished from zero. 

• On average, a 1 percentage point in- 
crease in the percent of students who left 
for a charter school has an effect on Eng- 
lish Language Arts (ELA) test scores of 
0.02 standard deviations, which is statis- 
tically significant at 10%. Two observa- 
tions are warranted. Eirst, this is a small 
effect, also noted in the report. Second, it 
is on the margin of statistical signific- 
ance (i.e., it is not statistically significant 
at the more commonly applied signific- 
ance level of 5%). 

• On average, the effects of competition 
are larger on students from the lower 
quartiles of the achievement distribution. 
That is, lower-achieving public school 
students experience slightly larger ef- 
fects in response to a 1 percentage point 
increase in the percent of students who 
left for a charter school, but never larger 
than 0.08 standard deviations (still a rel- 
atively small effect). 

The report’s conclusions are generally modest 
and aligned with the statistical findings just 
described. As stated by the author, the report 

find[s] some evidence that increases 
in the competition that a traditional 



New York City public school faces 
from charter schools for students 
leads to an increase in the ELA pro- 
ficiency of students who remain in 
public schools. Competition from 
charter schools also benefits students 
with very low prior math proficiency 
(p. 8). 

The report also correctly notes that the statis- 
tical findings do not necessarily imply that 
increases in the measure of competition 
cause test scores to rise (although they are 
correlated in some instances). Eor example, 
the effects of increasing competition could 
reflect the test score influences of correlated 
(but omitted) variables such as peer quality 
(p. 9). Does the competition variable exclu- 
sively proxy the competitive pressures facing 
public school administrators? Or, does it 
proxy other variables, such as a sudden shift 
in the peer environment because of departing 
students? This is an important issue to re- 
solve, since either could be responsible for 
the observed pattern of test score effects. This 
issue is revisited in the following sections. 

III. The Report’s Use oe 
Research Literature 

Much of the quantitative research literature 
on charter schools can be divided into two 
sets of studies, based on their research ques- 
tions. The report very briefly reviews each 
set, focusing on the second. 

The first set of studies examines the impact 
of transferring from a public school to a 
charter school on the outcomes of transfer- 
ring students. Put simply, does attending a 
charter rather than a public school have a 
causal effect on the outcomes of a transfer- 
ring student? (By causal, I mean that charter 
students’ achievement isn’t better or worse 
simply because of pre-existing differences 
between students, such as family income.) 
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The best recent studies have sought to an- 
swer this question by analyzing charter 
school lotteries. When charter schools are 
over-subscribed, states typically require a 
randomized admissions process. The win- 
ners of charter school lotteries constitute the 
treatment group, and the losers (who attend 
the alternative of their choice) constitute a 
similar control group. Any differences in 
outcomes are ascribed to “charter school 
quality”, broadly construed to include dif- 
ferences in school resources, teacher quality 
and practices, peer quality, and other va- 
riables. The report’s author admiringly cites 
unpublished work by Hoxby, Murarka, and 
Kang, which conducted a lottery study in 
New York City public schools.^ These au- 
thors find positive effects of attending a 
charter school on the test scores of transfer- 
ring students, but the validity of these prom- 
ising results has been questioned in a recent- 
ly released review."^ 

A second set of studies pursues a very dif- 
ferent research question: what is the effect 
of charter schools on students who remain in 
public schools. Since the majority of stu- 
dents will never transfer to charter schools, 
it is interesting to ask whether they may 
benefit indirectly. The basic theoretical pre- 
diction is that public schools facing greater 
competition from charter entrants will face 
incentives to increase their productivity and, 
hence, student outcomes. Whether this ac- 
tually occurs is an empirical question, and it 
has been studied in several U.S. states, using 
rich datasets very similar to the longitudinal 
New York City data of this report. As meas- 
ures of charter school competition, these 
studies usually employ measures of local 
charter school availability, such as the num- 
ber of charter schools within a defined ra- 
dius of a public school. The report cites a 
recent review article by Gill and Booker that 
reviewed the collected studies and found a 
mix of zero and slightly positive effects.^ 



The mixed pattern of findings is corrobo- 
rated in another review article of the charter 
school competition literature by different 
authors.^ The pattern of findings appears to 
vary substantially based on the particular 
state being analyzed and the statistical me- 
thod applied. 

A more comprehensive look at the literature 
on this topic would have alerted readers to 
the substantial methodological challenges of 
identifying the effects of competition. 

Prior studies of charter school competition 
share an objective with this report: to identi- 
fy the causal effect of competition on stu- 
dent outcomes. However, these studies (and 
the present report) face a technical challenge 
that the report mentions in the conclusions 
but does not carefully review. Whenever 
public schools face an increasingly competi- 
tive environment, “competition” is invaria- 
bly accompanied by a re-sorting of students 
across public, charter, and even private 
schools. As stated by Hsieh and Urquiola, 
who addressed the problem in an evaluation 
of Chile’s private school voucher system, 

A central issue in measuring the ef- 
fect of school choice is that it can 
simultaneously affect both schools’ 
productivity and the extent of sorting 
or stratification observed in the edu- 
cational system. If [school choice] 
influences schooling outcomes 
through both of these channels, then 
it is nearly impossible to disentangle 
their respective magnitudes. 

Imagine that a new charter school opens and 
draws students from a local public school. 
Indeed, the proportion of public students 
exiting to a charter school is the measure of 
competition used in the report. A student 
exodus from the public school increases 
competitive pressures, to be sure, but it si- 
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multaneously removes a group of students 
from the public school (we needn’t make any 
judgment about whether this is “cream- 
skimming” of high-ability students, or wheth- 
er exiting students are lower- ability). If the 
remaining public students are affected by the 
student departures — either positively or nega- 
tively — then so-called “peer effects” on stu- 
dent outcomes are confounded with the prod- 
uctivity-enhancing effects of competition. 

In the language of evaluators, two treatments 
occur at the same time. As a consequence, it 
is difficult to ascertain whether the observed 
impact of “competition” is driven by produc- 
tivity enhancements in public schools, or 
simple changes in the quality of peer envi- 
ronments. In addition to the present report, 
the critique also applies to other studies that 
attempt to identify the causal effect of com- 
petition of public school outcomes. The next 
sections will consider how the empirical re- 
sults can be more judiciously (and cautious- 
ly) interpreted in light of this critique. 

IV. Review of the Report’s 
Methods 

Measuring Competition 

The measure of charter school competition 
is the proportion of a public school’s total 
enrollment that left for a charter school at 
end of the last school year. The calculation 
of this measure is facilitated by the availa- 
bility of year-to-year, longitudinal data on 
each student’s enrollment in a public or 
charter school. 

The report is unclear about two specific fea- 
tures of this variable. First, it is probable that 
some students left a public school to attend a 
private school or an open-enrollment public 
school, rather than a charter school. The re- 
port’s measure does not account for such 
students (except in the calculation of the 



subsequent year’s school enrollment, the 
denominator of the competition variable). It 
is hard to imagine why public school admin- 
istrators would vigorously compete in re- 
sponse to losses from a charter school, but 
not a neighboring private or public school, 
since both signify lost market share. (Simi- 
larly, one imagines that Dell is equally con- 
cerned about consumer defections to HP, 
Toshiba, Apple, and other competitors.) In 
light of this, the analysis should control for 
related measures of competition from pri- 
vate and other public schools and ascertain 
whether the current results are robust. 

Second, it is not clear how the analysis treats 
potential student inflows from charter, pri- 
vate, or other public schools. It appears that 
such inflows, if they exist, are ignored or 
coded as a zero outflow to charter schools, 
although this decision is not well-explained. 
If a departing student poses a competitive 
threat in the simple framework of this study, 
then an arriving student, presumably, dulls 
the threat and exerts a countervailing effect 
on test scores. Again, it would be worthwhile 
to calculate alternative measures and assess 
the robustness of the main empirical findings. 

Data and Variables 

The report uses a rich dataset on students in 
New York City schools. The main sample 
includes up to three years of students’ test 
scores from 2007, 2008, and 2009, and is li- 
mited to students attending public schools. 
Longitudinal data from administrative data 
systems are an increasingly prominent ingre- 
dient in education policy analysis; indeed, 
researchers have employed similar longitu- 
dinal data sets to study the competitive ef- 
fects of charter schools in Texas, California, 
North Carolina, Florida, and elsewhere. 

Like other studies, the report limits the sam- 
ple to students attending public schools. The 
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report argues against including students at- 
tending charter schools because the measure 
of charter school competition — the propor- 
tion of students leaving for charter schools — 
is less relevant within a sample of charter 
schools (p. 7). But, following the limited 
theory on this topic, it is reasonable to as- 
sume that: (1) charter schools compete with 
each other; and (2) charter schools compete 
with public and private schools. Since these 
competitive responses also affect student 
achievement, it is worth thinking about how 
to comprehensively measure their effects. In 
the present framework, a natural starting 
point would be to use a richer set of competi- 
tion measures that reflects outflows and in- 
flows from all types of schools; and enlarge 
the sample of schools to include every pub- 
licly-funded NYC school with achievement 
data, including charter schools. 

Empirical Approach 

With the variables and sample thus defined, 
the report attempts to identify the statistical 
association between the competition variable 
and test scores. As the report observes, it 
would be foolhardy to estimate the simple 
correlation between these variables and as- 
cribe it to the causal effect of competition. 
Suppose 10% of public school students de- 
part, and that these 10% happen to be low 
achievers. In this case, the compositional 
changes wrought by “competition” would 
mechanically increase the school’s test 
scores. One approach is to control for ob- 
served characteristics of students of stu- 
dents, such as gender and socioeconomic 
status, but this is likely to omit or miss key 
attributes of students that are correlated with 
the both the competition measure and test 
scores. As a sensible alternative, the authors 
control for “student fixed effects.” The un- 
derlying statistical methods are complex, but 
the procedure amounts to controlling for all 
student variables that do not vary across 



time, whether observed or unobserved. Natu- 
rally, this controls for observed and easily- 
obtained variables like gender and race. More 
importantly, it controls for unobserved and 
hard-to-measure student attributes (such as 
innate ability) that do not vary across time. 

V. Review of the Validity 
OF THE Findings And 
Conclusions 

By applying a student fixed-effects analysis, 
does the report succeed in identifying the 
causal effect of competition? To answer this 
question, we must ask whether there are un- 
observed and omitted determinants of test 
scores that vary across time, and whether 
such variables are correlated with the meas- 
ure of competition. In such a case, the meas- 
ured effects of competition could reflect the 
impact of correlated but unobserved va- 
riables. There are two plausible scenarios 
under which this might occur. 

First, the departure of students for charter 
schools (and a concomitant reduction in 
school enrollments) lowers class sizes in the 
short run. Suppose that 10% of students de- 
part for a charter school. In most public 
schools, it is unlikely that small enrollment 
changes will lead to short-run reductions in 
the number of classroom sections. If the av- 
erage class size is 20, then a departure of 
10% of students could lower class sizes by 2 
students in the following year. Of course, 
public schools may adjust personnel levels 
over several years, especially if enrollment 
declines are large and persistent. But, in the 
one-year time period over which test scores 
effect are measured, an increase in “compe- 
tition” is likely confounded with a reduction 
in class size. 

Second, the departure of students will modi- 
fy the peer environment in public schools. 
The inclusion of student fixed effects con- 
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trols for variables that are time-invariant. 
However, a sudden departure of students 
represents an equally sudden (and time- 
varying!) modification of school peer 
groups. Thus, a short-run increase in “com- 
petition” is confounded with a change in the 
composition of school peer groups, with un- 
certain implications for student achieve- 
ment. Achievement could decline if depart- 
ing peers are especially motivated or high- 
achieving, and it could increase if departing 
peers have the opposite qualities. Neither 
effect would occur if remaining students are 
simply unaffected by attributes of their 
school or classroom peers, although the best 
empirical evidence confirms the importance 

o 

of peer quality in determining test scores. 

In its conclusion, the report concurs with 
this analysis, noting that 

it is possible that some of the posi- 
tive relationship between competi- 
tion from charters and student profi- 
ciency in traditional public schools is 
explained by changes in peer quality 
rather than improvements in public 
schools’ effectiveness (p. 9). 

The report suggests that the small, positive 
ELA test score effect could be partially ex- 
plained by an improving peer environment 
in public schools (termed “reverse cream- 
skimming”). In fact, the small positive effect 
on ELA scores could also be explained by a 
large negative effect of peer sorting, and a 
large countervailing effect of productivity- 
enhancing competition (or a large negative 
effect of competition, and a large counter- 
vailing peer effect!). Without further analy- 
sis, there is no way to unpack the “black 
box” of the empirical results. However, giv- 
en the rich longitudinal data, the author 
missed an opportunity to do so. For exam- 
ple, one could calculate (time-varying) 
measures of peer quality or class size in each 



school. By including such variables as addi- 
tional controls in the statistical analysis, one 
could determine whether the measured ef- 
fects of the competition proxy are attenuated 
(or perhaps made even stronger). 

In summary, the report’s estimates of com- 
petitive effects should be interpreted with a 
dose of caution. To be clear, this does not 
mean that the current study is uninteresting, 
or a poor contribution to the literature on 
charter schools. But, it is not consistent with 
extant economic literature to infer that the 
partial correlations summarized in section II 
are the causal effect of increasing competi- 
tion from charter schools. Rather, the esti- 
mated effects represent the net impact of 
several changes occasioned by the sudden 
departure of students: (1) an increase in 
competitive pressures and public school res- 
ponses; (2) an immediate decline in class 
sizes; and (3) an immediate modification in 
the quality of school peers of remaining stu- 
dents. Without further study, it is impossible 
to disentangle these effects using the re- 
ported empirical findings, and the report 
does not attempt to do so. 

The report’s text describes the results accu- 
rately and, for the most part, provides a 
guarded interpretation. Unfortunately, this is 
not reflected in the report’s optimistic title 
(“Everyone Wins”). A longer, dryer, and 
more descriptive title might be: “Some win 
and some lose, but, on average, they slightly 
win — though mainly on the ELA test and 
with caveats about the causal interpretation 
of the effects.” 

VI. Usefulness of the Report 
FOR Guidance of Policy 
AND Practice 

The literature on charter schools has focused 
on estimating their effects on students who 
transfer from a public to a charter school. 
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Much of the attention of policy-makers and 
scholars continues to revolve around the im- 
portant, but relatively narrow question of 
whether charter schools “work” in raising stu- 
dent achievement of a select group of students. 

While important, these studies cannot ad- 
dress important general-equilibrium policy 
questions about competition. To its credit, 
the present report addresses these questions, 
although it cannot conclusively answer 
them. As a contribution to the body of re- 
search on this topic, the study has several 
strengths, shared by other recent studies: 

• It is based on a high-quality, longitudinal 
dataset of student achievement. 

• It uses appropriate statistical methods, 
namely student fixed-effects regressions. 

• It contributes to an established literature 
that finds roughly consistent effects of 



“competition” that are often zero or 
slightly positive, depending on the state 
and method. 

The study also has important weaknesses: 

• The fixed-effects methods, while appro- 
priate, are not a methodological panacea; 
they cannot control for several potential 
biases. 

• As a result, the measured effects of 
competition could also reflect the influ- 
ence of shifting peer quality, declining 
class size, or other unobserved variables. 

While the report notes some of these weak- 
nesses, it does not make a serious attempt to 
assess their validity. Thus, we still cannot 
conclusively understand whether “everyone 
wins” (or why) when charter schools com- 
pete in urban schooling markets. 
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